Skip to content

Use one help text per metric name across collectors - #571

Open
antoinedeschenes wants to merge 1 commit into
prometheus-community:masterfrom
antoinedeschenes:fix-consistent-metric-help
Open

antoinedeschenes wants to merge 1 commit into
prometheus-community:masterfrom
antoinedeschenes:fix-consistent-metric-help

Conversation

@antoinedeschenes

@antoinedeschenes antoinedeschenes commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #361

When two collectors report the same metric name with different help texts, the registry rejects the scrape with has help X but should have Y. Under the default HTTPErrorOnError handling, the whole /metrics request returns 500, so every metric from the exporter goes missing, not just the conflicting one.

This happens in two ways:

  • Google rewrites a built-in description, as in Exporter fails when stackdriver updates their help strings #361. Projects can briefly report the old and the new description at the same time.

  • Projects define the same custom metric type with different descriptions. Custom and auto-created descriptors are per project, and their description is whatever the writer set. We hit this with a Spark metric published by Dataproc Serverless batches into two projects: one project's descriptor had an empty description, the other had Auto-created custom metric.. Every scrape failed for the half hour the batches ran each day:

    error gathering metrics: 278 error(s) occurred:
    * [from Gatherer #2] collected metric stackdriver_cloud_dataproc_batch_custom_googleapis_com_spark_executor_bigquery_metrics_source_bytes_read {...} has help "" but should have "Auto-created custom metric."
    

    This is not specific to one metric. Comparing four of our projects, every auto-created Dataproc descriptor (about 70 metric types under custom.googleapis.com/spark/, agent/, cpu/, disk/, memory/ and network/) has an empty description in one project and Auto-created custom metric. in the other three, so the default text seems to depend on when the descriptor was created. Any exporter covering both kinds of project fails whenever both have live series.

Change

A new MetricHelpStore keeps the first help text seen for each metric name and returns it for every later descriptor with that name.

  • The Runtime creates one store and passes it to every collector it builds, through a new MetricHelpStore field on MonitoringCollectorOptions. All collectors in one exposition therefore serve the same help text.
  • A collector built without a store gets one of its own, so the signature of NewMonitoringCollector is unchanged and existing embedders keep working. Embedders that register several collectors in one registry should pass a shared store.

The help text served for a name stays fixed for as long as that metric keeps being reported. If Google updates a description, the new text only shows after a restart, or after the metric has gone unreported for an hour. I think that's the right trade: help is metadata, and the alternative is a failed scrape.

  • Why first seen wins, even when it is empty. Collectors run concurrently, so replacing a stored text during a scrape would put two help texts in the same exposition, which is the failure this fixes.
  • Bounded size. The store holds one entry per metric name, not per series. Names not asked for during an hour are dropped the next time a new name is added, so churning custom metric types or rotating projects do not grow it without bound.

Testing

  • TestCustomMetricWithDifferentDescriptionsPerProjectGathers reports one custom metric type from two projects with different descriptions and gathers them through a registry. Without a shared store it fails with the same has help "Auto-created custom metric." but should have "" error we saw in production. With it, the gather returns one family with both series, under the first description seen.
  • TestRuntimeCollectorsShareMetricHelpStore checks that every collector a Runtime builds uses the runtime's store.
  • Unit tests cover the store on its own: concurrent callers get one text, unused names are dropped while live ones are kept, and the zero value is usable.

I also ran both builds against the Cloud Monitoring API, with two projects whose descriptors for custom.googleapis.com/cpu/utilization differ only in description, and --monitoring.metrics-prefixes=custom.googleapis.com/cpu:

Build /metrics Series served Gather errors
master 500 0 829 has help "Auto-created custom metric." but should have ""
this branch 200 830 (829 from one project, 1 from the other) 0

On this branch both projects report stackdriver_monitoring_last_scrape_error 0, and the family is served with one help text, the one from whichever collector reported it first.

🤖 Generated with Claude Code

Each project collector builds its metric descriptors from its own
project's metric descriptor. Projects can describe the same metric type
differently, custom and auto-created metrics in particular, and the
registry then rejects the gather with "has help X but should have Y",
which fails the whole scrape.

Add a MetricHelpStore that keeps the first help text seen for each
metric name, for as long as that name keeps being reported. The Runtime
creates one and passes it to every collector it builds through
MonitoringCollectorOptions, so all collectors in an exposition serve
the same help text. A collector built without a store uses one of its
own. Names not reported for an hour are dropped, which bounds the store.

Fixes prometheus-community#361

Signed-off-by: Antoine Deschênes <antoine.deschenes@linux.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@antoinedeschenes

Copy link
Copy Markdown
Contributor Author

cc @kgeckhart @SuperQ

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Exporter fails when stackdriver updates their help strings

1 participant